Modelling
Data science building blocks
types of machine learning [1]
Linear models
The basic shape of a linear regression is a line (in \(\mathbb{R}^2\)):
\(Y = W X + b\)
- \(Y = \{y_1, \dots, y_m \}\) are labels
- every label \(y_i\) has a set of features \(X=\{x | x \in \mathbb{R}\}\)
- \(b\) is a starting value, also called the “bias”
Linear models
The basic shape of a linear regression is a line:
\(Y = W X + b\)
- \(X = \{x_1, \dots, x_m\}\) where every \(x\) is the value of a feature and we have \(m\) features.
- \(W\) is a set of weights. Instead of \(W X\) we could also write: \(w_1 x_1 + w_2 x_2 + \dots + w_m x_m\).
- By writing capitals \(W\) and \(X\) we typically mean that they are collections of numbers, not a single number.
Classification
For classification models, we can still use a linear model. Only now we try to separate values, instead of getting them on the line.
Classification
The model is still: \(Y = W X + b\)
- \(Y\) is still a label, but typically want to end up with a class like “yes”, “no”, “heartattack”, “buy”, etc.
- so you will need some conversion, like:
\[Y = \begin{cases} \text{"buy" if } y \geq 0\\ \text{"sell" if } y \geq 0\\ \end{cases} \]
Multiple models
classifiers = [
('linear', SVC(kernel='linear')),
('kernel', SVC(C=1.0)),
('rf', RandomForestClassifier(n_estimators=100)),
('bayes', GaussianNB()),
('gauss', GaussianProcessClassifier()),
('kNN', KNeighborsClassifier(n_neighbors=5)),
('tree', DecisionTreeClassifier(criterion='gini'))
]
for i, (name, clf) in enumerate(classifiers):
clf.fit(X_train, y_train)
result = cross_val_score(clf, X_test, y_test, cv = 5)
mu = np.mean(result)
stderr = np.std(result)/np.sqrt(cv)
plt.scatter(i, mu, label=name)
plt.errorbar(i, mu, yerr=stderr)Hyperplanes
It’s a bird… It’s a plane…
- For classification, our outcome are discrete classes (e.g. “yes” or “no”)
- For regression, our outcome is a real number (e.g. 1.4 or 26.834, denoted as \(\mathbb{R}\))
We can use a linear model for both cases. We call these models “hyperplanes”: they have one dimension less than the ambient space.
With classification, we want the data to be separated by the hyperplane. With regression, we want to points to get as close as possible to the hyperplane
Support Vector Machines
Which model would you prefer?
Support Vector Machines
We would want to have some sort of safety margin, a maximal “dead zone” with a minimal amount of “violations”.
Support Vector Machines
If our labels are \(y = \{1, -1\}\) we would want \(y (w x_i + b) \geq 1\) to be true.
But let’s say \(x_1\) is a little bit inside the margin.
And \(x_2\) might be completely on the wrong side of the margin. How to account for these “violations” of our “safe border”?
Support Vector Machines
We assign a “slack value” \(\xi_1\) and \(\xi_2\) to these errors. These values are what is needed to “correct” the errors.
In the end, we will add \(C \Sigma_i \xi_i\) to the loss function. \(C\) is a value we pick (e.g. 1), and we simply sum up all the “slack” values.
We will then prefer the model with the lowest loss.
Non-linear models
A lot of data is non-linear. This means we would need a curved hyperplane.
One trick to do this is the kernel-trick, which is commonly used with Support Vector Machines, which we touched upon in the short history.
Deep learning has found another trick, which will be covered in the deep learning course.
The Kernel trick
Kernels
Kernels
Too complex models
Using basis expansion might seem a nice trick. But it can be much too powerful…
With too much variables, your model will become too complex very quickly.
The image on the left is a polynomial basis expansion to \(x^{100}\). As you can see, the model is way to complex.
Hyperparameters
